Extended reality-based user interface add-on, system and method for reviewing 3d or 4d medical image data

ABSTRACT

The invention relates to a system (1) for reviewing 3D or 4D medical image data (2), the system (1) comprising (a) a medical review application (MRA) (4) comprising a processing module (6) configured to process a 3D or 4D dataset (2) to generate 3D content (8), and a 2D user interface (16); wherein the 2D user interface (16) is configured to display the 3D content (8) and to allow a user (30) to generate user input (18) commands; (b) an extended reality (XR)-based user interface add-on (XRA) (100); and (c) a data exchange channel (10), the data exchange channel (10) being configured to interface the processing module (6) with the XRA (100); wherein the XRA (100) is configured to interpret and process the 3D content (8) and convert it to XR content displayable to the user (30) in an XR environment (48); wherein the XR environment (48) is configured to allow a user to generate user input (18) events, and the XRA (100) is configured to process the user input (18) events and convert them to user input (18) commands readable by the MRA (4). The invention also relates to an extended reality-based user interface add-on (100), a related method for analysing a 3D or 4D dataset (2), and a related computer program.

FIELD OF THE INVENTION

The invention relates to a system for reviewing 3D or 4D medical imagedata, an extended reality-based user interface add-on, a method foranalysing a 3D or 4D dataset, and a related computer program.

BACKGROUND OF THE INVENTION

For the visualisation of internal structures of a human or animal body,in particular of organs and tissues, medical imaging modalities, such asX-ray computed tomography (CT), magnetic resonance imaging (MRI) andultrasound, are generally used to acquire two-dimensional (2D),three-dimensional (3D) or four-dimensional (4D) images (the fourthdimension being time) of the interior of the body. These medical imagesplay an important role in the planning of surgical interventions.Especially in interventions in which a medical implant is implanted, forexample an artificial heart valve, a comprehensive presentation of theanatomy allowing the most accurate planning possible is crucial for thesuccess of the treatment.

3D image data sets may be either acquired directly, or they may beproduced by scanning a number of adjacent 2D image slices, usually in aregular pattern, and combining them, e.g. by a computer, into avolumetric grid. Each volume element (voxel) of the 3D image is assignedthe grey-scale value of the pixel in the respective 2D image slice,which represents the region around the voxel coordinates.

From a 3D image, 2D image planes can be generated in any orientation byinterpolating the voxel/pixel values closest to the desired 2D imageplane. This process is called Multi Planar Reconstruction (MPR) and isoften used to visualise a 3D image dataset.

In order to enable physicians to examine 3D images of the internalstructures of a body on a two-dimensional screen, a number of techniqueshave been developed. Volume rendering, for example, is a set oftechniques used to display a 2D projection of a 3D discretely sampleddataset, such as a medical 3D image. To render a 2D projection of the 3Ddataset, one needs to define a camera position relative to the imagevolume. One also needs to define properties of each voxel, such asopacity and colour. This is usually defined using an RGBA (red, greenblue, alpha) transfer function that defines the RGBA value for everypossible voxel value. In order to simplify the rendering process, oftena threshold is used, wherein voxels having a value below the thresholdare treated as having a value of zero, in order not to clutter the viewonto the important structures. Different techniques, such as volume raycasting, splatting, shear warp or texture-based volume rendering may beused.

While in many cases the image data sets are three-dimensional in nature,the analysis of clinical data and surgical planning is conventionallydone on two-dimensional screens. Although there are already verysophisticated software solutions available for most clinicalapplications, the navigation within 3D datasets is still mostly done bymanually shifting, tilting and rotating MPR planes within a volume, orrotating flat 2D representations of 3D volume-rendered models.Generating a conceptual mental 3D model from these MPR planes or volumerenderings planes not only requires a lot of experience and good spatialimagination, but always carries the risk of wrong clinical decisions dueto misinterpreting depth information or anatomical correlations.Advanced colouring and shading is often used to generate and support theimpression of depth on the two-dimensional screen. Furthermore, 2D inputdevices, such as a computer mouse, that are conventionally used, areoptimized for the manipulation of 2D objects. Thus, working on 3Dobjects via MPR planes is not only unintuitive in some cases, especiallyfor people new to the field, but can often also be quite time consuming.

An approach to improve workflows concerning medical 3D image datasets isthe use of 3D representations such as virtual reality (VR) or 3Dscreens. The application of such technologies has become more and morefeasible during recent years. For the first time, the technicaldevelopments in the field of virtual reality enable end-customers to buyaffordable, off-the-shelf hardware and software. Examples of this arethe well-received HTC Vive or the Oculus Rift VR headset, which havebeen available since 2016 and have launched a new era in the gaming andentertainment industry. Current developments will make this technology(especially augmented reality) available to the broad public in the nextyears (Apple AR Kit, Apple AR glasses in 2020, Oculus Quest, Vive Focusetc.)

The advantages of these VR headsets are the (partial or complete)immersion in a virtual environment and the real 3D representation of 3Dcontent. In addition, the highly intuitive control via tracked handcontrollers allows a very natural interaction with virtual objects (e.g.grabbing, rotating).

As neither standard 2D screens nor an VR environment are optimal for theentire medical workflow, virtual reality interfaces have been used incombination with 2D interfaces in order to benefit from the advantagesof both systems. For example, A. Bornik et al. have proposed such acombination in “A Hybrid User Interface for Manipulation of VolumetricMedical Data,” 3D User Interfaces (3DUI'06), Alexandria, Va., USA, 2006,pp. 29-36, doi: 10.1109/VR.2006.8. Here the user can switch between astereoscopic screen using shutter glasses and the 2D screen of a tabletPC, wherein both applications share a large portion of their programmedcode. A similar approach is proposed by V. Mandalika et al. in “A Hybrid2D/3D User Interface for Radiological Diagnosis”, J Digit Imaging 31,56-73 (2018), doi: 10.1007/s10278-017-0002-6, where a user can switchbetween a 2D display using mouse and keyboard and a 3D zSpace displayusing a stylus pen for input. Sørensen et al. propose in “A new virtualreality approach for planning of cardiac interventions”, ArtificialIntelligence in Medicine 22, 2001, 193-214 an approach to 3Dvisualization of MR data It is based on a tool for interactive,real-time visualization of 3D cardiac MR data sets in the form of 3Dheart models displayed on virtual reality equipment.

However, implementing such combinations of virtual reality and 2Dinterfaces is computationally complex, and requires special equipment,which is likely to be a problem for many medical facilities that have towork on a limited budget and have limited space available. Replacingexisting equipment with these new options or providing space and moneyfor additional equipment may often not be an option.

OBJECT OF THE INVENTION

It is, therefore, an object of the invention to provide a system and arelated extended reality-based user interface add-on, method andcomputer program that facilitates and optimizes the navigation withinand analysis of 3D datasets, while at the same time minimizingadditional expenses associated with such a new system and maintainingthe benefits of conventional systems.

SUMMARY OF THE INVENTION

This object is met or exceeded by a system for reviewing 3D or 4Dmedical image data according to claim 1, an extended reality-based userinterface add-on according to claim 11, a method for analysing a 3D or4D dataset according to claim 12 and a computer program according toclaim 15. Advantageous embodiments are set out in the dependent claims.Any features, advantages or alternative embodiments described herein inrelation to the claimed system are also applicable to the other claimcategories, in particular the claimed extended reality-based userinterface add-on, the method, and the computer program and vice versa.

According to the invention, a system for reviewing 3D or 4D medicalimage data is provided, the system comprising

(a) a medical review application comprising a processing moduleconfigured to process a 3D or 4D dataset to generate 3D content, and a2D user interface, wherein the 2D user interface is configured todisplay the 3D content generated by the processing module and to allow auser to generate user input commands;(b) an extended reality (XR)-based user interface add-on;(c) a data exchange channel operatively coupled to the processingmodule, the data exchange channel being configured to interface theprocessing module with the extended reality (XR)-based user interfaceadd-on;

-   -   wherein the data exchange channel is adapted to direct the 3D        content generated by the processing module to the extended        reality-based user interface add-on, and    -   wherein the extended reality-based user interface add-on is        configured to interpret and process the 3D content and convert        it to XR content displayable to the user in an extended reality        environment;    -   wherein the extended reality environment is configured to allow        a user to generate user input events, and the extended        reality-based user interface add-on (100) is configured to        process the user input events and convert them to user input        commands readable by the medical review application; and    -   wherein the data exchange channel is adapted to direct user        input commands from the extended reality-based user interface        add-on to the medical review application.

The advantage of the inventive system is that the extended reality(XR)-based user interface add-on is comparatively independent of anddecoupled from the medical review application, which may be an alreadyexisting software solution for reviewing, in particular displaying andanalysing, 3D medical images, such as for example one of the advancedTOMTEC® 4D clinical application packages (CAPs), such as 4D LV Analysis,4D RV Function, 4D MV Assessment or 4D Cardio-View. If at all, onlyslight modifications have to be made to these powerful andwell-established medical software products, in order to allow theexchange of data, e.g. the communication of volume data (3D or 4Ddatasets), graphical objects, coordinate systems and user interactions,with the extended reality (XR)-based user interface add-on. Inparticular, the medical review application does not need to be able todirectly drive or control XR hardware, such as VR goggles, since thisfunctionality is situated in the XR-based user interface add-on. TheXR-based user interface add-on may be configured to allow theapplication of various different XR hardware types, or there may beseveral XR-based user interface add-ons available, each for a differentXR hardware type. Furthermore, the XR-based user interface add-on can beused for different medical review applications, which again simplifiesimplementation. Therefore, the system of the invention allowsXR-interaction for dedicated workflows of an existing medical product(medical review application) without complete re-implementation.Instead, the XR-based user interface add-on provides a remote controland view add-on that provides all functionality for various XRworkflows, yet without any re-implementation of the business logic ofthe original medical product. By using the system of the invention, theuser can change between 2D and XR at any time and experiences a seamlessintegration of the two different user interfaces. The XR content isalways in perfect sync with the medical review application. Accordingly,workflows can be realised in which some measurements or other workflowsteps are carried out using the 2D user interface, and others, thatrequire better spatial overview of the image, may be carried out usingthe XR environment.

The system of the invention is preferably implemented as a softwareproduct comprising a medical review application and an extendedreality-based user interface add-on, connected by a data exchangechannel. Such software may be installed on any computer or other digitalprocessing device. The system may also be embodied in hardware, inparticular a computer having a processing unit, a data storage anddevices to allow user input and output, as explained in more detailbelow.

The 3D or 4D medical image data may be any 3D or 4D image datasetsgenerated by means of a medical imaging modality, such as CT, MRI orultrasound. It is also possible that the system can process several 3Dor 4D datasets at the same time. Processing the 3D or 4D dataset may inparticular comprise a rendering of the 3D or 4D dataset into 2D or 3Dimage or video data. 3D image or video data may in particular be twosets of 2D image or video data adapted to be provided for a 3Dstereoscopic view, as used by the extended-reality user interface. The3D or 4D medical image data preferably depicts a part of the human oranimal body, such as an internal organ, a limb or part thereof, a head,brain, fetus etc. The system of the invention is particularly useful inreviewing 3D or 4D medical image data of the moving human heart, inparticular 4D image data, i.e. a time series of 3D image datasetsacquired over at least one heartbeat. Images of this kind may beacquired with transesophageal ultrasound probes and are often used toanalyse the function of the heart, in particular the pumping action ofthe ventricles, as well as the functioning of the heart valves, and forplanning heart surgeries, such as replacing a heart valve with animplant. In the planning of such surgical interventions, the bestpossible visualization of the complex and dynamic valve structure isessential. The medical review application may also be referred to as 3Dsoftware product, since it is designed to review 3D or 4D medical imagedata, wherein reviewing comprises e.g. visualizing the images and theanatomical structures contained therein, allowing the user to navigatethrough these images, for example by manually shifting and tilting MPRplanes through the volume, providing 2D representations of the 3D or 4Ddataset by techniques such as volume rendering or surface rendering, andanalysing the dataset for example by fitting 3D models to the anatomicalstructures. The medical review application may also allow a user to takemeasurements within the 3D or 4D dataset, or checking the goodness offit of a model. All of these functionalities may be part of the medicalreview application, which may for example have the functionalities ofthe TOMTEC® CAPs mentioned above.

The medical review application comprises a processing module which isconfigured to process the 3D or 4D dataset to generate 3D content. 3Dcontent preferably is data having spatial coordinates, in particular 3Dcoordinates in the image space of the 3D or 4D image dataset. Thus, the3D content may comprise 3D image data (volume data), graphical objects,such as a mesh representing a model or graphical primitives, or an MPRplane, which generally consists of a 2D image, referred to as the MPRtexture, and the position and orientation of the MPR plane in 3D imagespace. The processing may comprise calculations performed upon the 3D or4D dataset, which generate either a modified 3D or 4D dataset, avisualization of the 3D or 4D dataset (such as volume rendering) or mayinvolve analysing the 3D or 4D dataset, such as fitting a model toanatomical structures, performing measurements within the dataset,calculating an MPR texture etc. The processing of the 3D or 4D datasetmay be based on user input commands, for example relating to theshifting, rotating or tilting of an MPR plane, the correction of amodel, or 3D mouse positions indicating the spatial coordinates ofmeasurement points or anatomical landmarks set by a user. The 3D contentmay be passed from the processing module to the 2D user interface and/orthe XR-based user-interface add-on in the form of 2D or 3D (i.e.stereoscopic) image or video data, obtained for example by renderingmethods from the 3D or 4D dataset, or by calculating an MPR planethrough the 3D or 4D dataset.

Furthermore, the medical review application also comprises a 2D userinterface, which is configured to display the 3D content generated bythe processing module and to allow a user to generate user inputcommands. The 2D user interface is preferably a non-XR-based userinterface, meaning it does not involve representations in extended orvirtual reality. Preferably, the 2D user interface comprises a window(also referred to as diagnostic region) for displaying images, as wellas a graphical user interface (GUI) which enables the user to interactwith the application. The GUI usually comprises various buttons,sliders, and/or numerical input fields, which may be either on a screento be operated on by a cursor, or may be implemented in a separate userinput device. The 2D user interface is usually already available as partof the medical review application and allows a user to generate userinput commands by interacting with the GUI and/or by marking specificpoints, lines or regions on the displayed image in the diagnosticregion.

In addition to this stand-alone medical review application, the systemof the invention comprises a data exchange channel operatively coupledto the processing module of the medical review application, which isconfigured to interface the processing module with an extendedreality-based user interface, which is preferably operatively coupled tothe data exchange channel. Herein, the term “Extended Reality” (XR) ismeant to cover all of virtual reality (VR, complete immersion in avirtual environment), augmented reality (AR, the user sees the realworld around him with virtual objects placed therein), and mixed reality(MR, virtual objects interact with real objects, e.g. real objects mayobstruct the view on virtual objects). Thus, an XR environment is onethat allows a user to view the 3D content stereoscopically, i.e. eacheye sees a slightly different image, resulting in a “real” 3Drepresentation of 3D content, which is then termed “XR content”.Accordingly, orientation within a 3D dataset is much more intuitive andsimpler when using the XR-based user interface add-on, also referred toas XRA. Moreover, the XR-based user interface add-on may allow a user tointeract with virtual objects, in particular the displayed XR content,i.e. by grabbing and rotating using tracked hand controllers. This is avery intuitive way of interaction.

Accordingly, user input events may be generated by the user within theextended reality environment, such as deforming a 3D mesh, which e.g.represents a model of an anatomical structure, making annotations orplacing a graphical object representing e.g. a device, such as animplant. Such user input events may be generated by moving tracked handcontrollers (XR controller) and at the same time actuating a button onthe controller. Accordingly, an XR controller is like a 3D mouse. In theXR environment, i.e. in the scene that is presented to the user whenusing the XR-based user interface add-on, the user will see thedisplayed XR content, and possibly also user interface elements (UIelements), which he can actuate using the XR controller to for examplechange the settings concerning the display of the XR content such ascontrast or brightness, or to indicate the start of a certain step inthe workflow, e.g. a certain measurement or interaction with a virtualobject.

The role of the XR-based user interface add-on is to process such userinput events, if the user chooses to generate any, to convert them touser input commands, which are readable by the processing module, and todirect them through the data exchange channel to the processing module.For example, the XR-based user interface add-on may process a movementof an XR controller from a first 3D point to a second 3D point into theuser input command “move currently grabbed object from first point tosecond point” and direct this command to the processing module.

The 3D content generated by the processing module will be directed tothe XR-based user interface add-on through the data exchange channel.Preferably, this will be done at any time while the processing module isactive, i.e. the data exchange channel is adapted to direct the 3Dcontent generated at any time by the processing module to the XR-baseduser interface add-on. It may also be done only during a connectionbetween the processing module and the XR-based user interface add-on,i.e. when the XRA is active. The latter is configured to interpret andprocess the 3D content and convert it to XR content displayable to theuser in the XR environment. This conversion will e.g. comprise thegeneration of the two slightly different views of the 3D content to bepresented to each eye for stereoscopic viewing. The conversion may inparticular comprise the adaptation of the 3D content to the momentaryhead position and/or viewing perspective, as detected by the VR hardwareused, for example by applying perspective distortion. Thereby, the 3Dcontent is converted to XR content.

The data exchange channel accordingly allows the exchange of variouskinds of data and information between the medical review application(MRA) and the extended reality-based user interface add-on (XRA). Onecould say that this exchange allows remote control of the otherwiseisolated 3D software product (MRA) by an external application (XRA).Therein, 3D content, in particular pre-defined 3D content such as a 3Dmesh generated by the processing module e.g. through segmentation of the3D or 4D medical image data, or 3D measurements, are to be transferred.Further, preferably user interface elements (such as the number of acurrent frame, settings concerning the display of the 3D content, etc.)may be synchronized between the two applications.

Conversely to the prior art, in the present invention the more complexcalculations, such as the generation of an MPR plane (MPR texture),generation of a measurement, segmentation, generation of a model of ananatomical structure, and deforming such a model, are all carried out bythe MRA, while the XRA will have as few tasks as possible, and just asmany as necessary in order to support a clinical workflow step. Suchworkflow step may for example be the review and modification of a modelrepresented by a graphical object such as a mesh, wherein the modelmight be a mitral valve segmentation, by means of the stereoscopic XRview and the 18 degrees of freedom given by an XR headset and two XRcontrollers. Therein, the XRA will allow a user to generate user input,for example “take this point in the model and move it one cm to theright”, but the actual modification of the model will be made by theMRA. The amended graphical object representing the model will betransferred back to the XRA. Because the XRA follows simpleinstructions, in particular to display certain 3D content, such as “. .. display object 1 at position 1 and object 2 at position 2”, the XRA isindependent of the MRA and may be used with many different MRAs.

The data exchange channel transfers data between the MRA and the XRA.When information is directed from the MRA to the XRA, the XRA processesthe information to allow its display in the XR environment. Informationsent from the XRA to the MRA, such as user input commands, may beprocessed and displayed by the MRA.

According to an embodiment, the MRA comprises a data interface for theXRA, wherein the data interface is operatively coupled to the dataexchange channel, and is configured to allow the exchange of simplifiedand standardized operating actions and data between the MRA and the XRA.Accordingly, the data interface is placed between the back end (MRA) andthe front end (XRA) of the system according to the invention, anddefines the type of data that can be exchanged between the two. Forexample, it allows the exchange of the simplified and standardizedoperating actions. Operating actions may, for example, comprise a 3Dmouse position and/or user input commands.

In an embodiment, at least a part of the data interface ismessage-based, and the messages are defined through user interfaceelements (UI elements). Preferably, at least some of the UI elements maybe modified by the user by actuating corresponding buttons or sliders onthe GUI and possibly in the XR environment, other UI elements may bechanged by other user input, such as mouse/controller events. Messagesexchanged may refer to a unique UI element defined by its unique ID(UID). The UI elements have also a distinct type, defining which kind ofvalue they transport. Such types may for example be string, Boolean,integer values, double values, etc. Thus, an exchanged UI element canhave one of the following events attached to the message:

UIevent_Value_Changed: This event is sent if the value of the parameterthe UI element is representing has changed. All UI elements may sendthis event. According to the invention, the direction of thecommunication may be in both directions, allowing synchronization of UIelements between MRA and XRA.

UIevent_Enable_Changed: By sending this event, the MRA tells the XRAthat a certain parameter is enabled or disabled. If a UI element is notenabled, no UIevent_Value_Changed event will be sent or accepted. Thistype of message is only sent to the XRA.

UIevent_Range_Changed: This event indicates that the valid range of avalue has changed. This type of message is only sent to the XRA and onlyvalid for UI elements that are in a defined range.

This embodiment is merely an illustration on how the data exchangebetween the XRA and the MRA may be implemented. It shows that MRA andXRA may be built independently, only a list of UI elements with thedescription of their purpose and usage, as well as possible 3D content,have to be defined.

Regarding the 3D content to be processed by the XRA, it is possible thatthe processing module will perform all processing of the 3D or 4Ddataset, so that the 3D or 4D dataset itself is not transferred to theXRA, but only respective volume renderings or MPR textures, togetherwith their 3D position and orientation, as well as other graphicalobjects such as meshes or graphical primitives and annotations.Alternatively, the 3D or 4D dataset itself may be transferred to theXRA, which in this embodiment is able to process it in order to generatestereoscopic renderings, in particular volume renderings. The XRA mayalso receive a data-reduced, simplified or data-compressed version ofthe 3D or 4D dataset.

According to an embodiment, the data interface is adapted tocontinuously synchronize corresponding user interface elements (UIelements) between the XRA and the MRA through the data exchange channel,wherein the corresponding user interface elements comprise at least oneof the value of the user interface element, and identifier of a selectedframe of a 4D dataset, settings concerning the display of the 3D and/orXR content, and/or a 3D mouse position. UI elements accordingly, may beany vector or parameter that relates to the display of the 3D and/or XRcontent. In some embodiments, also the position of a 3D/XR cursor may becontinuously synchronized between MRA and XRA. In other embodiments,such cursor position is not continuously synchronized. However, the userinput events in the XR environment generated with XR controllers, whichincorporate a 3D mouse position, are transferred from the XRA to the MRAwhen necessary, for example when a measurement is to be initiated or asegmentation, model or other graphical object is to be adapted. In someembodiments, the settings concerning the display of the 3D and/or XRcontent as well as the current frame of a sequence of 3D images (4Ddatasets) are continuously synchronized between MRA and XRA in bothdirections, so that the 3D content displayed to the user in the 2D userinterface is displayed in a comparable manner as XR content in the XRenvironment. The settings concerning the display of 3D/XR content mayfor example be brightness and contrast, as well as parameters relatingto a volume rendering of the 3D/XR content, such as threshold andopacity.

According to a preferred embodiment, the UI elements are synchronizedbetween MRA and XRA, wherein the MRA nevertheless is responsible formaintaining the values for the UI elements. This may imply that certainUI elements have pre-defined maximum or minimum values which are unknownto the XRA. Thus, the XRA may communicate a user input command that a UIelement be increased. However, if the UI element has already reached itsmaximum value, the MRA will answer by re-sending the old value for theUI element. Thus, the synchronization of UI elements is managed by themedical review application, wherein the XRA can only communicate itswishes to change the UI elements, but the medical review applicationdecides whether these changes are made or not.

Accordingly, in an embodiment, the XRA is stateless, in that it does nothave a memory of user input commands transferred to the processingmodule through the data exchange channel. Also by this measure, the XRAcan be kept slim with as little internal intelligence as possible. Inpreferred embodiments, the XRA is only able to transfer user inputcommands and to receive commands regarding the display of 3D contentfrom the MRA, and preferably has no further functionality. In anembodiment where the XRA is stateless, it can easily be plugged-on tovarious medical review applications, and require little or nomemory/buffer.

According to a preferred embodiment, a stream of data comprising 3Dcontent and optionally user input commands are exchanged through thedata exchange channel by means of a data connection. Preferably, thismay be a standard data connection. For example, the communicationbetween MRA and XRA can be via a TCP/IP socket connection. For example,messages may be exchanged as strings with a defined format like“<UID>|<event>|<value>”. For example, the message “thresholdtissue|value changed|125” sets the threshold for volume rendering to125. In this setup, the MRA and the XRA could run in differentprocesses, such as on different processing units. Another way tocommunicate is via a DLL interface. The application can be integratedinto a C++ DLL. This DLL provides a function “set UI element” (UIDevent, value) for the communication between XRA and MRA. In this setup,MRA and XRA are running in the same process, but in separated threads.

According to an embodiment, the 3D content generated by the processingmodule may comprise a rendering of the 3D or 4D dataset, wherein theextended reality-based user interface add-on is configured to adapt, inparticular distort in perspective, the rendered 3D content based on atleast some of the user input and/or a user's current viewingperspective. The rendered 3D content is in particular image or videocontent created via the rendering of the 3D or 4D dataset. Inparticular, the extended reality-based user interface add-on may beconfigured to adapt the viewing perspective more quickly than the rateof a data stream of rendered image or video data from the medical reviewapplication to the extended reality-based user interface add-on. E.g.,the data stream may be updated ca. 10 times per second, while theXR-based user interface add-on may be configured to update the viewingperspective 30 to 120 times per second, and thereby provide a realisticXR environment to the user, which for example reacts quickly to headmovements by the user wearing a VR headset.

According to an embodiment, responsive to a “switch user interface”command generated by a currently-active user interface among the XRA andthe 2D user interface, the processing module is adapted to stopresponding to user input commands from said user interface and to startresponding to user input commands from the other user interface.Thereby, the XR environment may be seamlessly integrated into themedical review application, without imposing any additional software onthe user, except the XRA. During a routine 2D workflow executed on the2D user interface, the user can thus give a user input command, e.g.click one button “view in XR”, and put on his headset to enter the3-dimensional XR environment, e.g. a VR space. He can then execute apart of the workflow in XR, preferably a part that is cumbersome to doin 2D. Afterwards, he may put down the headset and can instantlycontinue to work on his 2D workflow on the 2D user interface.

According to an embodiment, the 3D content generated by the processingmodule comprises at least one of the 3D or 4D dataset itself, an updatedor data-compressed version of the 3D or 4D dataset, a rendering of the3D or 4D dataset, a particular frame of a 4D dataset, an MPR texturegenerated from the 3D or 4D dataset, a graphical primitive, a 3D or 4Dmodel of an object, such as an anatomical structure, a mesh, a text ofan annotation, and/or a number indicating a measurement. Thus, theprocessing module may transfer volume data, such as the 3D or 4D datasetor an updated version thereof, for example a version that has beensegmented, or cropped to cut off the non-relevant tissue, to the XRA.The XRA will then process the volume data to generate XR content, forexample by rendering the same, such as volume rendering or surfacerendering, wherein the rendering will result in two slightly differentviews for each eye for stereoscopic viewing. In another embodiment, theprocessing module will perform the rendering of the volume data itselfand transfer the rendered views to the XRA. The processing module mayalso transfer a particular frame of a 4D dataset, i.e. one of a timesequence of 3D datasets. In another embodiment, the processing modulewill once transfer the complete 4D dataset, which is, in thisembodiment, buffered by the XRA during the complete review session. Theprocessing module in this embodiment need not transfer a particularframe, but only the identifier or number of the respective currentframe. Another type of 3D content may be a 2D image generated from the3D or 4D dataset by multiplanar reconstruction, a so-called MPR texture,together with its position and orientation. The XRA may then generatethe MPR plane in the XR environment at the correct position andorientation.

Another type of 3D content is a 3D or 4D model of an object, typicallyof an anatomical structure, such as a heart valve or a heart chamber, orthe model of an implant, for example a model of an artificial heartvalve. The model may be in 4D, i.e. it may change over time, for exampleover one heartbeat. A model is preferably a simplified, parametricrepresentation of the modelled object. The model will typically be asurface model, i.e. it is constructed of one or several, possiblymoving, surfaces. In a preferred embodiment, the 3D or 4D model of ananatomical structure is represented by a mesh, i.e. it is defined by aset of points in space, which span a triangular mesh. An importantapplication of the invention is also the displaying of implants as a 3Dor 4D model. By allowing the user to move around the 3D or 4D model ofan implant in the XR environment, while at the same time displaying avolume rendering of the 3D or 4D dataset, e.g. the heart, the user isable to very efficiently and correctly place the implant and plan thesurgery. Further, 3D content may also be an annotation, or rather thetext thereof, as well as its 3D position, allowing the XRA to displaythe annotation text at its correct position in space in the XRenvironment. Similarly, also a number indicating a measurement may betransferred. Further, 3D content may be a landmark position in the 3Dimage space. Another type of 3D content which may be transferred may betermed graphical primitive, which may be any standardized graphicalobject, such as a text window, a line, a point, or a graphical objectsuch as a triangle, a number of triangles, a sphere, etc. Also, diversedynamic graphical primitives may be transferred, e.g. to display(textured) surface models, measurements and annotations/landmarks. In anembodiment, the XRA is able to buffer such graphical primitives in orderto achieve a smooth dynamic display.

In an embodiment, during establishment of a connection between theprocessing module and the XRA, the processing module is adapted totransfer a temporal and spatial reference system and at least one of the3D or 4D dataset, a user interface element and optionally configurationsettings to the XRA. This may be done through the data interface of theMRA. This will serve to initialize the XRA and enable it to communicatesmoothly with the processing module. The configuration settings caneither be deployed together with the XRA, or can be transferred by theprocessing module during establishment of a connection, e.g. at thebeginning of a review session. This is especially preferably if the XRAis potentially used for more than one medical review application. Theconfiguration settings to be transferred during establishment of aconnection may comprise a unique application identifier (applicationname and a version of the medical review application), special startoptions/flags, and/or exchange configuration, such as IP address, portand exchange folders. The configuration settings may also compriseprotocol configuration (protocol version, allowed types of commands),configuration settings of the UI elements (which buttons and menus)and/or style for data objects (line width, line colour, etc).Accordingly, fundamental data may be transferred when starting a session(establishment of a connection), which provides a reference system forthe further data exchange between MRA and XRA during a session. Thetemporal and spatial reference system may comprise a common temporalreference system for the exchange between MRA and that allows to convertbetween frames and times. It serves as a reference system to definephase loops and time stamps. A spatial reference coordinate system isfurther exchanged that allows to convert between mm and pixelcoordinates in image space. It serves as a spatial reference system e.g.to position XR objects in relation to the 3D volume. Moreover, the 3D or4D dataset may be passed at the start of the session and, in anembodiment, is treated as invariable during a session.

According to an embodiment, the extended reality-based user interfaceadd-on (XRA) is adapted to transfer at least one of a 3D mouse position,a position and orientation of an MPR plane, a screenshot, and/or amodified value of a user interface element, through the data exchangechannel to the processing module, during a connection between theprocessing module and the XRA. Thereby, the XRA can act as a “remotecontrol” to the medical review application, like a 3D mouse.Accordingly, user input events such as a click on a XR controller at acertain position in space, i.e. a 3D mouse position, may be transferredto the MRA as a user input command. Such a command may for exampleindicate that a 3D model is adapted, by grabbing a point on thedisplayed mesh and dragging it to one side. However, the XRA does notmaintain the model itself, but only transfers the 3D mouse positions andrespective user interactions, e.g. the 3D position of a first click, anda 3D position of a second click, and the information that these userinput events relate to the modification of the 3D model. The processingmodule will translate these user input commands into a modification ofthe model. Similarly, the XRA may transfer the desired position andorientation of an MPR plane in the form of user input commands. However,the processing module will do the calculations to calculate a new MPRtexture, relating to the transferred position and orientation. The newMPR texture will be transferred back to the XRA. In an embodiment, alsoa screenshot generated in the XR environment may be transferred from theXRA to the processing module, in order to document an executed workflowstep. Moreover, the XRA may transfer UI elements, or modified valuesthereof, as described above. In preferred embodiments, the UI elementsare continuously synchronized between the MRA and the XRA. This kind ofdata may be termed transient data, as it is data that is exchangedcontinuously during a review session. In a preferred embodiment, thetransient data may be exchanged using a technique termed user interfacelibrary (UIL), which is used to decouple user interaction from thebusiness logic of the medical review application. When a UIL connectionis established using a TCP socket connection, the MRA and XRA cancommunicate using a set of shared UI elements, such as the selectedframe, as well as various settings concerning the display of the 3D orXR content, e.g. the threshold and transparency for volume rendering, aswell as brightness and contrast for MPR planes, a pen type for editingof 3D or 4D models, and settings for regulating which objects (volumedata, renderings and models) are displayed.

According to an embodiment, the XRA is configured to be used with XRhardware via an XR operating system, wherein the XR hardware inparticular comprises an XR headset and XR controllers. In a preferredembodiment, the XRA is configured to be used with commercially availableXR hardware, such as the HTC vive® or the Oculus Rift® VR headset and VRcontrollers. This XR hardware already comes with a XR operating system(driver software), and the XRA is configured to communicate with the XRoperating system of the XR hardware and to process e.g. user input likehead movements or XR controller interactions, and to direct the views ofthe 3D content generated in the XRA onto the two screens of the XRheadset. This may be achieved by the XRA using a standard API(Application Programming Interface), such as Open XR, to communicatewith the XR operating system. These XR/VR standards are commonly knownand allow to use many different XR headsets for the extendedreality-based user interface add-on. Preferably, the XR hardwarecomprises also two XR controllers, which can be used as 3D mouse,allowing the user to e.g. grab an object displayed in the VR environmentwith one hand and rotate/tilt/move it with the other.

According to a further aspect, the invention is directed to an extendedreality-based user interface add-on (XRA) configured to be operativelycoupled via a data exchange channel to a medical review applicationhaving a processing module configured to process a 3D or 4D dataset togenerate 3D content,

-   -   wherein the extended reality-based user interface add-on is        configured to interpret the 3D content received via the data        exchange channel and convert it into XR content in a data format        readable by an XR operating system of an XR hardware, wherein        the XR hardware in particular comprises a XR headset, such that        the XR hardware can display the XR content generated by the        processing module; and    -   wherein the extended reality-based user interface add-on is        configured to process any user input events received from the XR        operating system, in particular user input events generated        using XR controllers, and to convert the user input events into        user input commands readable by the medical review application,        and to transfer the user input commands via the data exchange        channel to the medical review application.

The XRA according to this aspect is preferably configured as explainedabove. It is a comparatively lean software solution that can be coupledwith a number of different medical review applications allowing toreview 3D or 4D datasets, as the XRA only requires the exchange ofcertain well-defined data. Such data exchange can be realized with adefined data interface to the MRA allowing the exchange of simplifiedand standardized operating actions and data via a data exchange channel.The extended reality-based user interface add-on may in particular beimplemented on a computer or digital processing device, having aprocessing unit, a data storage and devices to allow user input andoutput.

According to a further embodiment, the invention is also directed to amethod for analysing a 3D or 4D dataset, in particular of a human oranimal organ using the system as described herein. The method comprisesthe following steps:

-   -   processing the 3D dataset to generate 3D content on the        processing module;    -   optionally, the 2D graphical user interface displaying the 3D        content;    -   the data exchange channel directing the 3D content to the        extended reality-based user interface add-on and the extended        reality-based user interface add-on interpreting and processing        the 3D content and converting it to XR content displayable to a        user by XR hardware;    -   receiving user input commands on one of the user interfaces;    -   directing the user input commands to the processing module        directly from the 2D graphical user interface or via the data        exchange channel from the extended reality-based user interface        add-on;    -   the processing module processing the 3D content based on the        user input commands to generate modified 3D content,    -   directing the modified 3D content to the 2D graphical user        interface and the data exchange channel;    -   optionally, the 2D graphical user interface displaying the        modified 3D content; and    -   the data exchange channel further directing the modified 3D        content to the extended reality-based user interface add-on and        the extended reality-based user interface add-on interpreting        and processing the modified 3D content and converting it into        modified XR content displayable to a user by XR hardware.

These steps allow a user to interact with the 3D content, for example inorder to modify a model of an anatomical structure, or to takemeasurements, set landmarks or make annotations. This will result in amodified 3D content. According to the method of the invention, it ispossible for the user to make the user input in any one of a 2D userinterface or the XR user interface add-on, and to generate modified 3Dcontent therewith. The modified 3D content will be directed to the 2Duser interface and the data exchange channel, so that the user may viewthe modified 3D content either on the 2D user interface, or on the XRhardware. Accordingly, the invention is also directed to a workflow thatrequire some steps being implemented on the XRA and other steps on theconventional, 2D user interface. For the steps in the XRA, the 3Ddataset is rendered and the user is enabled to manipulate it and toprovide input events. However, the next workflow steps may take place onthe 2D user interface on the modified 3D content, based on the inputprovided by the user in the XRA. Thereby, it is possible to implementdifferent steps of a workflow on the user interface that is bettersuited for any given step. By using the disclosed XRA, the XR contentdisplayed by the XR hardware is always in sync with the medical reviewapplication.

According to a preferred embodiment, the method may comprise thefollowing steps:

-   -   processing the 3D dataset to generate a rendering of a 3D        dataset and a 3D model of an anatomical structure depicted by        said 3D dataset;    -   displaying the 3D model and the rendering via the extended        reality-based user interface add-on;    -   allowing a user to check the 3D model on the extended        reality-based user interface add-on and to provide user input        commands to adjust the 3D model;    -   the data exchange channel directing the user input commands from        the extended reality-based user interface add-on to the        processing module;    -   the processing module processing the user input to generate a        modified 3D model;    -   directing the modified 3D model to the 2D graphical user        interface, and the 2D graphical user interface displaying the        modified 3D model;    -   optionally allowing a user to perform additional analysis and/or        measurements on the modified 3D model in the 2D graphical user        interface.

Therein, the user may make optimal use of the XR environment in order tocheck a 3D model, for example a mesh representing the mitral valve. Itis also possible to try out different valve implants on the volumerendering of the mitral valve, preferably the dynamic representationthereof, in the XR environment.

According to an embodiment, the processing of the 3D dataset to generate3D content may comprise at least one of data-compressing the 3D or 4Ddataset, rendering the 3D dataset, volume rendering the 3D dataset,calculating an MPR texture of an MPR plane through the 3D dataset,segmenting the 3D or 4D dataset, generating a 3D or 4D model of anobject, in particular a medical device or anatomical structure,generating a graphical primitive, and/or taking a measurement responsiveto user input.

The invention is also directed to a computer program comprising programcode instructions, which, when executed by a processing unit, enablesthe processing unit to carry out the method disclosed herein, or toimplement the system according to the invention, or the extendedreality-based user interface add-on, according to the invention. Themethod may also be carried out on several processing units. Theprocessing unit or computational unit may be any processing unit such asa CPU (Central Processing Unit) or GPU (Graphics Processing Unit). Theprocessing unit may be part of a computer, a cloud, a server, mobiledevice such as a laptop, tablet computer, mobile phone, smartphone, etc.In particular, the processing unit may be part of an ultrasound imagingsystem.

The invention is also directed to a computer-readable medium comprisinginstructions, which, when executed by a processing unit, enable theprocessing unit to carry out the method according to the invention, orto implement the system or the XRA according to the invention. Suchcomputer-readable medium may be any digital storage medium, for examplea hard disk, a server, a cloud server, an optical or a magnetic digitalstorage medium, a CD-ROM, an SSD-card, an SD-card, a DVD or an USB orother memory stick.

According to a further aspect, the invention is directed to acomputational unit configured to implement the system according to theinvention. Such computational unit may comprise a processing unit asdescribed herein, as well as hardware to implement the 2D userinterface, such as a screen and a user input device such as a mouse,touch screen, track ball, etc. The computational unit is configured tobe used together with XR hardware as described herein, in particular anXR headset and XR controllers, in particular VR headset and VRcontrollers.

SHORT DESCRIPTION OF THE FIGURES

Useful embodiments of the invention shall now be described withreference to the attached figures. Similar elements or features aredesignated with the same reference signs in the figures. Differentembodiments shown are explicitly allowed to be combined unless notedotherwise.

FIG. 1 shows a state of the art 2D user interface on a screen that ispart of a medical review application;

FIG. 2 shows a schematic representation of the working principle ofvolume rendering of 3D images;

FIG. 3 shows a schematic illustration of a system according to anembodiment of the invention;

FIG. 4 shows a schematic representation of a system for reviewing 3D or4D medical image data according to an embodiment of the invention;

FIG. 5 shows a schematic illustration of the operational connectionbetween the medical review application and the XR-based user interfacevia the data exchange channel according to an embodiment of theinvention;

FIG. 6 shows a flow diagram of a method for analysing a 3D or 4D datasetaccording to an embodiment of the invention;

FIG. 7 shows a flow diagram of a method according to another specificembodiment of the invention in comparison with a conventional methodhaving the same purpose;

FIG. 8 shows a schematic view of the implementation of a systemaccording to the invention on a computer with corresponding hardware;

FIG. 9 shows a view from the perspective of a user when using theXR-based user interface according to an embodiment of the invention.

DESCRIPTION OF EMBODIMENTS

FIG. 1 shows a conventional state of the art 2D user interface 16 thatis part of a medical review application (MRA) 4, e.g. a TOMTEC® medicalreview application. In this case, the 2D user interface, comprises botha graphical user interface (GUI) 15 which enables the user to interactwith the application, and a region 17 showing a 2D representation of 3Dcontent 32 and which is termed the “diagnostic region” of the MRA 4. Inthe diagnostic region 17, the MRA 4 outputs a 2D representation of 3Dcontent 32, such as the 3D volume rendering 32. The GUI 15 and thecorresponding 2D representation of 3D content 32 of the diagnosticregion 17 may for example be displayed on a computer monitor or on thescreen of a tablet. User interaction, for example via a mouse or atrackball, which might comprise navigation or drawing of measurements,is generally handed through the diagnostic region 17 as well as the GUI15. In an embodiment, the diagnostic region 17 is a region, possiblydivided into several windows, in which the MRA 4 provides an opengraphics library (open GL) context for the rendering of the 2Drepresentation of 3D content 32. Common graphical user interface parts,like for example sliders for brightness or contrast, 3D filters andstart/stop buttons for controlling the display of time-dependent medicalimage data, are part of the graphical user interface 15, which surroundsthe diagnostic region 17 in the embodiment shown in FIG. 1 .

FIG. 2 shows a schematic representation of the working principle ofvolume rendering of a 3D image dataset 2. Volume rendering can forexample be executed in the form of direct volume rendering, inparticular volume ray casting. Looking at the left side of FIG. 2 , aray 35 is generated for each image pixel of the 2D image 32 that is tobe rendered.

This ray is directed, i.e. cast, through the 3D image volume 2containing 3D content. While the ray is passing through the volume,equidistant sampling points are selected. These sampling points aregenerally in between voxels and, therefore, the values of the samplingpoints are usually interpolated from the surrounding voxels. Thesampling points are then shaded, i.e. coloured and lit according totheir surface orientation and the location of a light source andcomposited along the ray of sight. This results in the final colourand/or brightness value for the processed pixel. Doing this once forevery pixel of the final 2D image will result in a 2D representation ofthe 3D content 32, as is shown on the left side of FIG. 2 . In order toobtain an XR representation of the 3D content, two differenttwo-dimensional images 34 are rendered from slightly differentperspectives, the two perspectives representing the left and the righteye of an observer. This is shown on the right side of FIG. 2 . The twoslightly different 2D images 34 are then projected into the left andright eye of a user respectively, thereby generating the impression of athree-dimensional object. Projecting different images into each of theuser's eye can for example be realized via an XR headset, VR glasses orby using a TV screen, a computer monitor or a projector screen with ashutter or polarisation technique and corresponding shutter orpolarization glasses. This is an example of how the XR-based userinterface add-on may convert 3D content 2 into XR content 34 displayableby XR hardware.

FIG. 3 shows a schematic illustration of a system according to anembodiment of the invention. A medical review application (MRA) 4comprises a data interface 4 a, through which the MRA 4 is connected toa data exchange channel 10. The data exchange channel 10 in turn isconnected to an XR-based user interface add-on (XRA) 100. Through thedata interface 4 a and the data exchange channel 10, the MRA 4 may sendcoordinates of measurement primitives and measurement values, which aregenerated at the MRA 4, to the XRA 100. The XRA 100 is operativelyconnected to an XR environment 48, in order to display XR content to auser. The XRA generates XR content from the 3D content, which the XRA100 receives from the MRA 4 through the data exchange channel 10. The 3Dcontent may for example comprise the coordinates of measurementprimitives and measurement values, as well as 3D or 4D images. It may bedisplayed statically, i.e. without animation, or dynamically, i.e. inthe form of an animation. In the case of a dynamic display, the XRA mayapply a buffering of objects to achieve a smooth display. In order tocommunicate with the XR hardware 40 of the XR environment 48, the XRA100 is configured to use the XR operating system 42 of the specific XRhardware 40, which is comparable to the driver of the XR hardware 40,and which may be commercially available. In more detail, the XRA 100 maybe configured to use an application programming interface (API) or asoftware development kit (SDK) of the XR operating system 42. The XRoperating system 48 then sends a stereoscopic image or severalstereoscopic images that represent the XR content to the XR hardware 40,in particular to an XR-based headset 44.

Optionally, the user 30 is allowed to provide user input 18 via an XRcontroller 46, for example by pressing or releasing a button, by movingthe controller and/or by pulling a trigger, or simultaneously doingseveral of these interactions. This user input 18 is registered at theXR operating system 42, which is adapted to transmit coordinates andinteractions of the user to the XRA 100. Hence, the user input 18 signalmay for example comprise coordinates describing the position of avirtual pointer controlled by the XR controller 46, a user command likepressing a button that conveys a certain meaning, e.g. the command totake a measurement or to manipulate a 3D model, and/or the timestamp ofthe user's action. Editing within one frame may be carried out whiledata is displayed statically. It is conceivable to have a play/pausefunction that allows the user 30 to switch between a dynamic and astatic mode. Furthermore, there might be a “previous frame”/“next frame”function to go through consecutive frames step by step. The XRA 100 isconfigured to process the user input 18 and direct updated information,like a 3D mouse position and interactions as well as a command to take ameasurement at a certain position and time, to the MRA 4 via the dataexchange channel 10 and the data interface 4 a. The MRA 4 is configuredto process this new information and generate an accordingly updated 3Dcontent. This updated content will again be directed to the XRA 100 viathe data interface 4 a and the data exchange channel 10, converted to XRcontent by the XRA 100, and be presented to the user via the XRenvironment 48 in the same manner as described before.

Advantageously, the user input commands comprising the user input 18that are submitted via the XRA are very basic and comparable to thecommands a user would submit via a computer mouse to a computer. Byintegrating common APIs like OpenXR, the XRA 100 can communicate with awide range of existing XR hardware 40 through the hardware's operatingsystem 42. Here, the XRA 100, on the one hand, prepares the 3D contentto be presented to a user 30 as XR content via the XR environment 48and, on the other hand, provides the means to “translate” user input 18via the XR hardware 40 into a language that can be understood andtransmitted by the data exchange channel 10. Because all the commandssubmitted this way are very simple and no processing of data other thanpreparing it for display to a user 30 is carried out, the XRA 100 itselfmay remain very simple. Through the use of a very universal language ofcommunication by the data interface 4 a and the data exchange channel 10that is compatible with many already existing MRAs 4, the XRA 100 can beused to provide an XR environment 48 for many different MRAs 4. The XRA100 thereby updates the MRAs 4 to not only have a 2D user interface butan additional XR user interface as well. By utilizing commonly availableXR hardware 40, the XRA thus provides an easily obtainable andcomparatively low-priced way of upgrading existing medical reviewingsystems.

FIG. 4 shows a schematic representation of a system for reviewing 3D or4D medical image data 2 according to an embodiment of the invention. Thesystem comprises a processing module 6 of an MRA 4, which is configuredto process 3D or 4D medical image data 2, which may be uploaded from adigital storage medium, in order to generate 3D content 8. This 3Dcontent 8 is then transferred to a 2D user interface 16 that displaysthe 3D content 8 on a 2D screen 54, for example a computer monitor or atablet screen, in the form of a 2D representation of 3D content 32. This2D representation of 3D content 32 may be observed and analysed by auser 30, who may provide user input 18 via a user input device of thegraphical user interface 18 b. The user input 18 is then directed to theprocessing module 6, which processes the user input 18. The processingmodule 6 and the 2D user interface 16 are part of an MRA 4. Such an MRA4 alone is known from the state of the art.

However, the system according to the invention furthermore comprises adata exchange channel 10 that is operatively coupled to the processingmodule 6 via a data interface 4 a and configured to interface theprocessing module with an additional user interface 14. In theembodiment shown in FIG. 4 , the data exchange channel 10 is operativelycoupled to an XRA 100, which in turn is coupled to an XR operatingsystem 42. User interface elements 24 of the MRA 4 and of the XRA 100are continuously synchronized via the data exchange channel 10. Byutilizing the XR operating system 42, the XRA 100 is coupled to XRhardware 40 comprising an XR headset 44 and an XR controller 46. Thedata exchange channel 10 is adapted to direct 3D content 8 generated bythe processing module 6 to the XRA 100, which in turn directs the 3Dcontent 8 in the form of XR content via the XR operating system 42 tothe XR headset 44. Finally, the XR headset 44 displays a 3Drepresentation of the 3D content 34 to the user 30. The displayedcontent may comprise for example static or dynamic textures on a surfacemodel and/or values for a measurement.

Advantageously, such a stereoscopic view may give the user 30 a bettergrasp of complex 3D environments and may even unveil a level of detailhardly possible in a 2D user interface 12. Furthermore, a 3Drepresentation of 3D content 34, i.e. of clinical data, is closer to asurgeon's view thus decreasing the gap between clinical procedure andanalysis. Additionally, as an XR view is in many cases more intuitiveand less abstract it may advantageously be used in various training andeducational contexts, as well as for helping to explain medicalconditions to a patient through visualization.

The user 30 is enabled to generate user input 18 via a user input deviceof the XR-based user interface 18 a, which in this case is an XRcontroller 46. It is also conceivable that the user 30 may use more thanone controller, e.g. one XR controller 46 in each hand. Each controllermay have a different task. For example, one controller might bededicated to an actual measurement, while the other controller is usedto hold and navigate MPR-planes and surface models. Alternatively, onecontroller might be used for rotating a whole scene, while the other oneis used for rotating the view around a fixed axis. In the case of anysuch user input 18, the system is configured to direct the user input 18via the XR operating system 42, the XRA 100, the data exchange channel10 and the data interface 4 a to the processing module 6, which in turnis configured to process the user input 18. Thereby, because XR utilizesthe human eye-hand coordination far more than mouse-based ortrackball-based approaches, a more intuitive navigation in medical 3D or4D medical image data 2 is made possible. This allows for more efficientand effective measurements and/or more direct input commands.Furthermore, it is also conceivable that the user 30 can switch betweendifferent measurement visualizations or between different datasets. Thismay be realized in connection with saving and loading bookmarks, e.g. ofUI elements or other settings and/or states of data analysis.

Additionally, it is conceivable that a presenter being an active user30, e.g., a presenter in a lecture on a congress, executes a workflow inan XR environment 48 while several passive observers can watch usingtheir own XR hardware 40. Alternatively, the role of the active user 30may be switched during a medical discussion among two colleagues, e.g.among two physicians.

The XR operating system 42 can also be seen as a driver software for theXR hardware 40 that is incorporated in or used by the XRA 100 tocommunicate with an XR hardware 40, i.e. with the XR headset 44 and theXR controller 46. Advantageously, the XR operating system 42 can be anapplication programming interface (API) such as e.g. OpenXR, whichsupports various different XR hardware devices. The system is adapted toallow the user 30 to switch between the XR-based user interface 14 andthe 2D user interface 16 of the MRA at any time. Therefore, the user 40can, for example, look at a 2D representation of 3D content 32 at the 2Dscreen 45 of the 2D user interface 16 in order to get an overview of themedical data, e.g. of an organ, and then switch to the XR-based userinterface 14, in order to have a more detailed and possibly moreintuitive look at the 3D content 8 via the 3D representation of the 3Dcontent 34. Next, the user 30 may issue user input commands 18 at theXR-based user interface 14 via the XR controllers 46, for example torotate the image or take some measurements. Afterwards the user 30 mayswitch back to the 2D user interface 16 of the MRA 4 to have a 2D lookat changes of the 3D content 8 issued by the processing module 6 due tothe previous user input 18 at the XR-based user interface 14. The user30 may then revise the 3D content 8 and possibly apply further changesvia the user input device of the graphical user interface 18 b.

FIG. 5 shows a schematic illustration of the operational connectionbetween the MRA 4 and the XR-based user interface 14 via the dataexchange channel 10 according to an embodiment of the invention. In thisembodiment, the MRA 4 and the XR-based user interface 14 share atemporal and spatial reference system 20 through the data exchangechannel 10. This temporal and spatial reference system 20 is preferablyexchanged during the initial connection between the MRA 4 and theXR-based user interface 14, when establishing a review session (“initialhandshake”). For example, it allows the conversion between volume framesand times and defines phase loops and time stamps. A phase loop may beunderstood as a temporal region within the complete 3D dataset, forwhich 3D measures or segmentations are created. While the 3D data maycomprise several heart cycles, it is beneficial in some cases to create3D measurements or segmentations of only one cycle or of only a part ofone cycle that is most interesting for the analysis of the heart or of apart of the heart, such as a mitral valve. A dynamic display of such aphase loop comprises animating over the phase loop. In this embodiment,these phase loops, and in particular the range of those phase loops, aswell as time stamps within such phase loops, are synchronized betweenthe MRA 4 and the XR-based user interface 14.

Furthermore in this embodiment, the MRA 4 and the XR-based userinterface 14 share a common coordinate system 20 via the data exchangechannel 10. It serves as reference system, for example to position 3Dobjects in relation to the 3D volume. Furthermore, the XR-based userinterface 14 comprises configuration settings 22 that are used during anestablished session with an MRA 4. The configuration settings maycomprise a unique application identifier, such as an application nameand an application version, special start options, an exchangeconfiguration (e.g. an IP address, a port and/or exchange folders), aprotocol configuration (e.g. a protocol version and/or allowed types ofcommands), a UI configuration and style options for data objects (e.g.line width and colour). These configuration settings 22 allow theXR-based user interface 14 to communicate via the data exchange channelwith an MRA 4 in order to receive and display 3D content 8, such as 3Dor 4D medical image data 2, a 3D or 4D model, or 3D primitives 26 fromthe MRA 4 through the data exchange channel 10. The configurationsettings 22 may further allow the XR-based user interface 14 to be usedwith more than one different MRA 4, and to adapt to the properties ofeach of the different MRAs 4. The configuration settings 22 may eitherbe stored on the XR-based user interface 14 permanently or they may betransferred via the data exchange channel 10 during the initialhandshake when initiating a session between the XR-based user interface14 and an MRA 4. Furthermore, it is provided that the MRA 4 and theXR-based user interface 14 share user interface elements 24, which arecontinuously synchronized via the data exchange channel 10. The userinterface elements 24 comprise a value of a user interface element, anidentifier of a selected frame of the 3D or 4D medical image data 2,settings concerning the display of the 3D content 8 such as a thresholdor transparency for volume rendering or brightness and contrast formultiplanar reconstruction planes, and/or a 3D mouse position.

During a session, 3D content 8 is directed from the MRA 4 via the dataexchange channel 10 to the XR-based user interface 14. The 3D content istypically generated by the processing module 6 and may compriserendering of the 3D or 4D medical image data 2, a particular frame of a4D dataset, an MPR texture generated from the 3D or 4D medical imagedataset, a 3D or 4D model of an object and/or a mesh. Furthermore, 3Dprimitives 26, a text of an annotation and/or a number indicating ameasurement may be transferred. In principle, it might be alsoconceivable to transfer the 3D or 4D medical image dataset directly viathe data exchange channel to the XR-based user interface 14 and renderit at the XR-based user interface 14. This necessitates renderingcapabilities of the XR-based user interface 14, but on the other handhas the advantage that the total data transfer between the MRA 4 and theXR-based user interface 14 is lower.

User input 18 at the XR-based user interface 14 is directed to the MRA4, in particular to the processing module 6, via the data exchangechannel 10. Preferably, the input is issued with a user input device ofthe XR-based user interface 18 a, e.g. an XR controller 46, but it mightbe also conceivable to use other input devices such as a computer mouseor keyboard. Furthermore, also a prompt to take a screenshot 28 may beissued at the XR-based user interface 14, which will be stored at theMRA 4 in order to review or print it later.

In summary, the XR-based user interface 14 may be configured to issueonly very basic commands via the data exchange channel 10 in combinationwith configuration settings 22, which allow for the XR-based userinterface 14 to be used with different MRAs 4. The XR-based userinterface 14 thus provides a very versatile, yet also simple solution toupgrade already existing MRAs 4 with comparatively low effort andexpenses.

FIG. 6 shows a schematic illustration of a method for analysing a 3D or4D dataset 2 according to an embodiment of the invention. The methodcomprises a step of processing the 3D or 4D dataset 2 on the processingmodule 6, in order to generate 3D content 8. This 3D content 8 is thendirected to the 2D user interface 16 and/or to the data exchange channel10, which in turn directs the 3D content to the XR-based user interface14. As a next step, the 3D content is then displayed on the XR-baseduser interface 14 and/or on the 2D user interface 16 of the MRA 4. Thisenables a user 30 to look at a 3D representation of 3D content 34(referred to as XR content) or at a 2D representation of 3D content 32,alternatively or successively. The user has then the option to issueuser input 18 either on the XR-based user interface 14 or on the 2D userinterface 16. User input 18 at the 2D user interface 16 will be directedto the processing module 6 directly, while user input at the XR-baseduser interface 14 will be directed to the processing module 6 via thedata exchange channel 10. The processing module 6 will process the 3Dcontent 8 based on the user input 18 and thereby generate modified 3Dcontent 8 a. This modified 3D content 8 a is directed to the 2D userinterface 16 and displayed at the 2D user interface 16 and/or directedto the data exchange channel 10, which further directs the modified 3Dcontent 8 a to the XR-based user interface 14 which also displays themodified 3D content 8 a. Optionally, the user 30 might again issue auser input 18 at either of the user interfaces 14, 18 which will bedirected either directly or indirectly via the data exchange channel 10,respectively, to the processing module for processing. Accordingly, thiscycle might be repeated as many times as it is necessary or useful forthe user 30 to achieve a desired result, in particular to complete aworkflow of reviewing medical image data.

FIG. 7 shows a schematic illustration of a method according to anotherspecific embodiment of the invention (right side) in comparison with aconventional method (left side) having the same purpose. The workflowsteps are shown one below the other, with the height of each stepindicating the amount of time (illustrated by arrow 260) required foreach step. In particular, a specific medical workflow for analysing apathologic mitral valve is shown, which is used to decide whichbioprosthetic valve size fits best as an implant that will be implantedat a surgical intervention. As a first step, the 3D dataset is loadedinto the MRA 242. Following this first step, initial landmarks areplaced for the segmentation of a mitral valve (MV) 244 on the 2D userinterface 16 of the MRA 4. After the second step, the workflow accordingto this embodiment of the invention 230 differs from the conventionalworkflow 220. In the conventional workflow 220, the third step is tocheck and adjust the segmentation on the 2D user interface 246 b, inparticular, on a 2D screen 54, with mouse and keyboard. For the workflowwith the XRA 230, the third step is to check and adjust the segmentationon the XR-based user interface 246 a. As tests have shown, the timeneeded for the third step in the workflow with the XRA 230 issignificantly lower than for the corresponding step in the conventionalworkflow 220. The following, fourth step is identical in both workflowsand consists of analysing the resulting MV parameters and selecting acorrect device size 248. In both cases this workflow step is carried outon the 2D user interface 16. Accordingly, the same amount of time isneeded for these four steps in both workflows. The fifth workflow stepdiffers again in that in the conventional workflow 220 additionalmeasurements are performed and/or the device position is checked andadjusted on the 2D user interface 250 b, while in the workflow with theXRA 230, additional measurements are performed and/or the deviceposition is checked and adjusted on the XR-based user interface 250 a.Again, it has been shown that significantly more time is needed for theconventional workflow step than for the corresponding workflow step withthe XRA 230. When the analysis is finished, in the final step 252 alarger amount of time 260 has passed in the conventional workflow 220than in the workflow with XRA 230.

In addition to the workflow steps described above, further workflowswith further workflow steps are conceivable, such as 4D analysis and/orassessment of the function of the left and right ventricle or of themitral valve via surface models, 4D cardio views for volume measurementsvia surface models, or analysis of 4D radiological ultrasound data, e.g.TomTec® SONO-SCAN. It has turned out that the workflow steps in an XRenvironment 48 are not only more efficient, i.e. faster, thanconventional 2D workflow steps, but they are also more effective andreliable by leading to a lower variability in measurement results.Furthermore, due to the more intuitive approach, a lower training timeof new users 30, e.g. physicians, is to be expected.

FIG. 8 shows a schematic illustration of the implementation of a systemaccording to the invention on a computer 300 with correspondinghardware. The computer may comprise a processing unit 302 and a digitalstorage medium 304, on which the system is installed as softwareproduct. The XR hardware 40, comprising XR controllers 46 and an XRheadset 44, is connected to the computer, on which the XR-based userinterface 100 is installed. At the same time an MRA 4 is also installedon the computer 300 and connected to hardware comprising user inputdevices of the 2D user interface 18 b and a 2D screen 54. A 2Drepresentation of 3D content 32 is displayed on the screen 54. The usermay switch between the XRA 100 and the 2D user interface 16 at any time.

FIG. 9 shows a view from the perspective of a user 30 when using theXR-based user interface 14 according to an embodiment of the invention.In this embodiment, the user 30 can see both a 2D representation of 3Dcontent 32 and a 3D representation of 3D content 34 at the same time inthe XR-based user interface 14. For example, a user 30 might rotate a 3Dmodel and thereby create new slices in a multi planar reconstruction. Onthe other hand, it is also conceivable to have different visualizations,e.g. of a model of a mitral valve, such as a wireframe model, a cutlinewith a plane and/or a cutline and a transparent (ghost) model in the XRenvironment.

The above discussion is intended to be merely illustrative of thepresent system and should not be construed as limiting the appendedclaims to any particular embodiment or group of embodiments. Thus, whilethe present system has been described in particular detail withreference to exemplary embodiments, it should also be appreciated thatnumerous modifications and alternative embodiments may be devised bythose having ordinary skill in the art without departing from thebroader and intended spirit and scope of the present invention, as setforth in the claims that follow. Accordingly, the specification anddrawings are to be regarded as an illustrative manner are not intendedto limit the scope of the appended claims.

REFERENCE SIGNS

-   1 system-   2 3D or 4D medical image data/dataset-   4 medical review application (MRA)-   4 a data interface-   6 processing module-   8 3D content-   8 a modified 3D content-   9 3D models-   10 data exchange channel-   12 user interface-   14 XR-based user interface-   15 Graphical user interface-   16 2D user interface-   17 Diagnostic Region-   18 user input-   18 a user input device of the XR-based UI-   18 b user input device of the 2D user interface-   20 temporal and spatial reference system-   22 configuration settings-   24 UI elements-   26 3D/graphical primitives-   28 screenshots-   30 user-   32 2D representation of 3D content-   34 XR representation of 3D content-   35 Ray-   40 XR hardware-   42 XR operating system-   44 XR headset-   46 XR controller-   48 XR environment-   54 2D screen-   100 XR-based user interface add-on (XRA)-   200 method-   202 processing the 3D dataset-   204 generate 3D content-   206 directing the 3D content-   208 display 3D content-   210 receiving first user input commands-   212 receiving second user input commands-   214 directing first user input commands to the processing module-   216 directing second user input commands to the processing module-   220 conventional workflow-   230 workflow with XRA-   42 loading 3D dataset into MRA-   244 placing initial landmarks for the segmentation of a mitral valve    (MV)-   246 a checking and adjusting the segmentation on the XR-based user    interface-   246 b checking and adjusting the segmentation on the 2D user    interface-   248 analysing resulting MV parameters & selecting correct device    size-   250 a performing additional measurements and/or checking and    adjusting the device position on the XR-based user interface-   250 b performing additional measurements and/or checking and    adjusting the device position on the 2D user interface-   252 finishing the analysis-   260 time axis of the workflow-   300 computer-   302 processing unit-   304 digital storage medium

1. A system for reviewing three-dimensional (3D), or four-dimensional(4D) medical image data, having a processing unit, a data storage anddevices to allow user input and output, and having an extended realityenvironment, the system comprising: a medical review applicationcomprising a processing module configured to process a 3D or 4D datasetto generate 3D content, and a 2D user interface, wherein the 2D userinterface is configured to display the 3D content generated by theprocessing module and to allow a user to generate user input commands;an extended reality (XR)-based user interface add-on; and a dataexchange channel operatively coupled to the processing module, the dataexchange channel being configured to interface the processing modulewith the extended reality, i.e. XR-based user interface add on; whereinthe data exchange channel is adapted to direct the 3D content generatedby the processing module to the extended reality-based user interfaceadd-on; wherein the extended reality-based user interface add-on isconfigured to interpret and process the 3D content and convert it to XRcontent displayable to the user in an extended reality environment;wherein the extended reality environment is configured to allow a userto generate user input events, and the extended reality-based userinterface add-on is configured to process the user input events andconvert them to user input commands readable by the medical reviewapplication; and wherein the data exchange channel is adapted to directuser input commands from the extended reality-based user interfaceadd-on to the medical review application.
 2. A system according to claim1, wherein the medical review application comprises a data interface forthe extended reality-based user interface add-on, wherein the datainterface is operatively coupled to the data exchange channel, and isconfigured to allow the exchange of simplified and standardizedoperating actions, in particular comprising a 3D mouse position and/oruser input commands, and data between the medical review application andthe extended reality-based user interface add-on.
 3. A system accordingto claim 1, wherein the medical review application comprises a datainterface for the XR-based user interface add-on, wherein the datainterface is adapted to continuously synchronize corresponding userinterface elements between the extended reality-based user interfaceadd-on and the medical review application through the data exchangechannel, wherein corresponding user interface elements comprise at leastone of a value of a user interface element, an identifier of a selectedframe of the 4D dataset, settings concerning the display of the 3Dand/or XR content, and/or a 3D mouse position.
 4. A system according toclaim 1, wherein the XR-based user interface add-on is stateless, inthat it does not have a memory of user input commands transferred to theprocessing module through the data exchange channel.
 5. The system ofclaim 1, wherein the 3D content generated by the processing modulecomprises a rendering of the 3D or 4D dataset, wherein the extendedreality-based user interface add-on is configured to adapt, inparticular distort in perspective, the rendered 3D content based on atleast some of the user input and/or a user's current viewingperspective.
 6. The system of claim 1, wherein, responsive to a “switchuser interface” command generated by a currently-active user interfaceamong the extended reality-based user interface add-on and the 2D userinterface, the processing module is adapted to stop responding to theuser input commands from said user interface and to start responding touser input commands from the other user interface.
 7. The system ofclaim 1, wherein the 3D content generated by the processing modulecomprises at least one of an updated or data-compressed version of the3D or 4D dataset, a rendering of the 3D or 4D dataset, a multi planarreconstruction, i.e. MPR, texture generated from the 3D or 4D dataset, agraphical primitive, a 3D or 4D model of an object, a mesh, a text of anannotation, and/or a number indicating a measurement.
 8. The system ofclaim 1, wherein the processing module is adapted to transfer a temporaland spatial reference system and at least one of the 3D or 4D dataset,an updated 3D or 4D dataset, a user interface element and/orconfiguration settings to the extended reality-based user interfaceadd-on during establishment of a connection between the processingmodule and the extended reality-based user interface add-on.
 9. Thesystem of claim 1, wherein the extended reality-based user interfaceadd-on is adapted to transfer at least one of a 3D mouse position, aposition and orientation of a multi planar reconstruction, i.e. MPR,plane, a screenshot, and/or a modified value of a user interfaceelement, through the data exchange channel to the processing moduleduring a connection between the processing module and the extendedreality-based user interface add-on.
 10. The system of claim 1, whereinthe extended reality-based user interface add-on is configured to beused with XR hardware via an XR operating system, wherein the XRhardware in particular comprises an XR headset and XR controllers. 11.An extended reality-based user interface add-on configured to beoperatively coupled via a data exchange channel to a medical reviewapplication having a 2D user interface and a processing moduleconfigured to process a 3D or 4D dataset to generate 3D content, whereinthe extended reality-based user interface add-on is configured tointerpret the 3D content received from the processing module via thedata exchange channel and convert it into XR content in a data formatreadable by an XR operating system of an XR hardware, wherein the XRhardware in particular comprises an XR headset, such that the XRhardware can display the XR content generated by the processing module;and wherein the extended reality-based user interface add-on isconfigured to process any user input events received from the XRoperating system, in particular user input events generated using XRcontrollers, and to convert the user input events into user inputcommands readable by the medical review application, and to transfer theuser input commands via the data exchange channel to the medical reviewapplication.
 12. A method for analysing a 3D or 4D dataset, inparticular of a human or animal organ, using a medical reviewapplication comprising a processing module and a 2D user interface, adata exchange channel operatively coupled to the processing module,wherein the data exchange channel is configured to interface theprocessing module with an extended reality-based user interface add-on,the method comprising the steps: processing the 3D dataset to generate3D content on the processing module; optionally, the 2D user interfacedisplaying the 3D content; the data exchange channel directing the 3Dcontent to the extended reality-based user interface add-on and theextended reality-based user interface add-on interpreting and processingthe 3D content and converting it to XR content displayable to a user byXR hardware; receiving user input on one of the user interfaces;directing the user input commands to the processing module directly fromthe 2D user interface or via the data exchange channel from the extendedreality-based user interface add-on; the processing module processingthe 3D content based on the user input commands to generate modified 3Dcontent, directing the modified 3D content to the data exchange channeland optionally to the 2D user interface; optionally, the 2D userinterface displaying the modified 3D content; and the data exchangechannel further directing the modified 3D content to the extendedreality-based user interface add-on and the extended reality-based userinterface add-on interpreting and processing the modified 3D content andconverting it into modified XR content displayable to a user by XRhardware.
 13. A method according to claim 12, the method comprising thesteps: processing the 3D dataset to generate a rendering of a 3D datasetand a 3D model of an anatomical structure depicted by said 3D dataset;displaying the 3D model and the rendering via the extended reality-baseduser interface add-on; allowing a user to check the 3D model on theextended reality-based user interface add-on and to provide user inputto adjust the 3D model; the data exchange channel directing the userinput commands from the extended reality-based user interface add-on tothe processing module; the processing module processing the user inputcommands to generate a modified 3D model; directing the modified 3Dmodel to the 2D user interface, and the 2D user interface displaying themodified 3D model; optionally allowing a user to perform additionalanalysis and/or measurements on the modified 3D model in the 2D userinterface.
 14. A method according to claim 12, wherein the processingmodule is configured to process the 3D or 4D dataset and generate 3Dcontent and/or XR content by at least one of data-compressing the 3D or4D dataset, rendering the 3D dataset, volume rendering the 3D dataset,calculating a multi planar reconstruction, i.e. MPR, texture of an MPRplane through the 3D dataset, segmenting the 3D or 4D dataset,generating a 3D or 4D model of an object, in particular a medical deviceor anatomical structure, generating a graphical primitive, and/or totaking a measurement responsive to user input.
 15. A computer programcomprising program code instructions which, when executed by aprocessing unit, enables the processing unit to carry out the methodaccording to claim 12.